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A6STBACT 

The Evaluation Services component of the Education 
^Service Center, in San Antonio, Texas, evaluated nine special 
XEducation programs conducted by the Center* It was decided to 
Waluate the actual impact of these reports, which examined the 
:prcgrams« major accomplishments. The thirty professional staff 
vinvoXved in the programs were surveyed. Fifty percent of the staff 
"^agreed that Evalua-tion Services had had a positive impact on their 
FEcigrams* However, their ratings and comments clearly reflected a 
need for specific improvement in the conduct of the evaluation 
process. Data are given on the findings of the survey and the actions 
taken by Evaluation Services. A copy of the survey instrument is 
appended. (JD) 
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During 1978-79 y the Evaluation Services component at Education 
Service Center, Region 20, evaluated nine Special Education programs 
conducted by the service center. These programs were designed to 
support LEAs In their efforts to provide appropriate educational pro- 
gramming for handicapped stiidents. The nine programs were funded at 
about 1.25 million dollars; somewhat over 5% of the funding was al- 
located to Evaluation Services. 

The evaluations were designed to answer a variety of questions 
specific to each program* s objectives and activities; however, the 
questions can be groxxped under the following general qxiestlons: 

(1) How lany hours of service were provided? 

(2) How many pupils of which type were served? 

(3) What was the service that was actually provided? 

(4) Which aspects of the service were of valiie to clients? 

(5) Which aspects of the service could be Improved? 

(6) Did students benefit from the services? 

(7) What services do clients need? 

Evaluation Services* reports addressed the major accomplishments 
of the programs. But, like many evaluators today, we did not trtist 
our Judgment of the impact of these reports on special education pro* 
grams* We decided to check out our perceptions by conducting a self— 
evaluation. 

Evaluation Services surveyed the thirty professional staff In- 
volved in these programs. Fifty percent of the professional staff 
agreed that Evaluation Services had had a positive impact on their pro- 
grams. However, their ratings and comments clearly reflected a need 
for specific improvement in" the conduct of the evaluation process. 
The following Using Evaluation Data pages document our 1978-79 find- 
ings and the actions we took. (Appendix A contains a copy of our 



^^^C ^'"^^y Instrument)* 
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A) The DOSt frequent coirint made jn response to open- 
ended items seeking ways for Evaluation Services 
to improve centered on increased understanding of 
people, programs, and the constraints. Typical 
conments were Evaluation Services had "very little 
concept of vhat ly work actually consists of", 
and that Evaluation. needs to "become more familiar 
vith... guidelines and program directives" and more 
"understanding of demands placed on staff". 



B) Almost as frequent were comments on Evaluation 
Services' perceived negativism. Special Education 
staff perceived a "feeling of surprise by Evaluation 
Services at positive Evaluation findings", and they 
requested "more positive wording in evaluations". 



C) Other is^ortant but less frequent comments included 
the following" 

Special Education staff and Evaluation Services 
should work together. Program staff requested that 
"Evaluation find out t^t kind of data Special 
Education wants and how they want it presented", 
and they also suggested that "project staff become 
aware of all aspects of evaluation". 



m 



Evaluation Services in developing Evaluation Plans 
for Special Education programs will formally budget 
time for getting to know staff and programs. This 
would include site visits to obsene the program, 
attendance at staff meetings, and background read- 
ind as well as discussion. 



Evaluation Services will report all data and sub- 
jective impressions in a less judgmental, more 
objective style. This applies to both written 
and oral communications. Any judgments made by 
Evaluations Services will be labelled as judg- 
ments of the evaluator working on the program. 

One, and only one, Evaluation Services staff member 
will be the primary "evaluator" for each special 
education project. All contact will go through 
this person. This staff member will ensure that 
program staff are involved in developing and have 
copies of the evaluation plan and that they under- 
stand each evaluation activity. Project staff will 
be kept informed of progress in the evaluation; 
they will share in decisions, reviewing drafts 
and instruments. 



Action means concrete policy, procedures, decisions, or assignments. 
5 "Nc action" nay be justified but should be explained for the record. 
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USING EVALUATION DATA 



FINDINGS 



Special Education staff suggested that Evaluation 
Services "eliminate useless evaluation at (the) 
end of workshops and evaluate participants' class- 
room use of the information". 



D) Analysis of responses to both the open-ended 
and the rating items revealed that Special 
Education staff felt the major benefits of eval-, 
uation were work on needs assessment and object- 
ives. 



E) Program improvement was seldon explicit in the 
comments on major benefits. 



F) Fifty percent (13) of the professional staff 
agree or strongly agree that Evaluation Services 
has had a positive impact on programs. About 
32Z (8) are neutral, and 18Z (5) disagree. 



ACTION 



Evaluation Services will limit workshop evaluation 
to (A) those specified in the Evaluation Plan, at|^ 
(B) those requested by Special Education persomi^^ 
as providing useful information to them (up to a 
certain limited number). 



Evaluation Services will continue to conduct an ^^^^ 
region-vide needs assessment focusing on involving 
to a greater degree Special Education staff and t^^Q^^. 
ing the findings to the LEAs. Evaluatiou Services 
staff will also continue their direct involvement ^ 
the development of progra objectives. 



Evaluation Services will (1) budget time for P^^ing 
use of evaluation data, ^2) focus on explicit fotin^. 
tive evaluation questio: j, and (3) maintain logs 
work with program staff on use of evaluation data, 



Evaluation Services has developed a set of o^jectj^g^ 
(see next page) for 1979-80 f - its work with spec^^j 
education programs; these represent Evaluation Set^^^^g' 
plan for improving its iMf&ct and provide for a 
evaluation of Evaluation Services at the end of th^ 
1979-80 program year. 



♦Action means concrete policy, procedures, decisions, or assignments, 
Ho action" may be justified but should be explained for the record. 
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FIVE YEAR OBJECTIVES FOR EVALUATION 
OF SPECIAL EDUCVTION PROGRAMS 

NOTE:.. These objectives have been s^c as long-range ones and designed to be 
'•tough to meet" but, once ttet» would assure us of the impact of Eval- 
uation Services* 

Objective 1: Special Education staff responsible for programs (coordinators, 

managers, and certain consultants) will have used information 
provided by Evaluation Services to improve special education 
programs. Accomplishment of this objective will be evidenced 
by at least two-thirds of the program staff interviewed citing 
"limited'' to "extensive" impact for at least three of the six 
areas measured per document by the in.'itruroent "Impa zt of Evalua* • 
tion Services" (Part II, J's 1-6) with at leajst one area per 
document rated as having, "marked" or "extensive" "inpact. The 
instrument will be administered by an external consultant for 
three select documents per program. * 

Objective 2: Special Education staff will report increased impact of Eval- 

uation Services. on special educatior programs* Accomplishment 
of this objective»will be evidenced by an increase of at least 
0.5 standard deviation units (mean from 3.5 (1978-79) to ^.0 
( future ) on the item "Evaluation Services has had a positive 
impact on ESC-20 special education j^rograi-s with which I am 
familiar" (Strongly disagree" to "strongly agree" on a 5-point 
scale with 3 « "neutral"). 

Objective 3: Evaluation Seryices staff will have better working relation- 

ships with ESC-20 special education staff. AcccTopli&h.-rent 

of this objective -will be indexed by at leasr tvo-rhircs of^ 

the special education staff responding "agree" or 'srrongly" 
agree" (A or 5 on a 5-point scale) to itesrs on the Ivaiusring 
Evaluation services survey addressing Evaluation Services' 
(a) lack of negativism (b) understanding of Special Ecccarion 
programs* staff, and constraints, (c) ability to vork prccuctively 
with program staff, and (d) desire to have Evaluation Ser\'ices 
for the program they are assigned. 

* Lack of resources prevented interviewing sufficient staff 
to measure this objective. 
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1979-80 RESULTS 

Having reviewed the 1978-79 findings, decided upon a course of action » and set 
1979-80 objectives, we were most eager to see the 1979-80 results. (See Instru- 
nent In Appendix 3). We were quite pleasantly surprised. First, with respect 
to objective 3. bettering our working relationships, which we considered ante- 
cedent to using evaluation results to improve a program: 

TABLE 1 

Ratings of Evaluation Services' Working Relationships with Special Education 



PERCENTAGE AND MEAN OF 
ITEM "AGREE" OR "STRONGLY AGREE' 



Work with me to find out what kind of 
information I want. 

Keep me informed of the progress of 
the evaluation. 



86 ^.3 
95 4.5 



Present findings in a positive, ^ ^ 

constructive way. 



Wbrk with me to interpret and use 
evaluation findings. 

Hake a positive effort to understand 
my program, my goals, and the con- 
straints I am facing. 



86 
81 

95 4.4 



Note — based on a N of 21 

Furthermore, hardly any of the respondents indicated on the second or third 
open-ended item that they have experienced any particular problems with 
Evaluation Services. However, three staff members did suggest that the method 
of charging for evaluation be changed; they felt that charging a set percen- 
tage of the program budget for evaluation is not the best way to fund those 
services, that charges should be based on the nature and amount of service 
needed. This was the only response made by more than one project staff member 
to these lt« 
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W€ also collected data on another antecedent to using evaluation results to 
Inprove a program: selecting and conmuni eating useful findings. 

TABLE 2 

Ratings of Evaluation Services* Communication of Potentially Useful Findings 



PERCENTAGE AND MEAN OF 
IT™ " AGREE*' AND "STRONGLY AGREE" 



Evaluation Services* written reports highlight 

those findings that could lead to improveme«*t 

of our services. 9q 

Evaluation Services* staff highlight in 
discussion those findings that could lead to 

improvement in our services. 85 4.4 

Note — based on an N of 21 ^ 

Second, with respect to objective 2, program staff reports of the impact of 
Evaluation Services: 

TABLE 3 

Ratings of Evaluation Services Impact on Special Educatio : Programs 



PERCENTAGE AND MEAN OF 
"AGREE" AND "STRONGLY AGREE" 
1978-79 1979-80 
Z X Z X 



The presence of Evaluation Services has had 
a pcsitlve impact in improving programs 

with which I am familiar. 50 3.5 75 



Based on my experiences with Evaluation 
Services, I believe Evaluation is a nec- 
essary and important service. 



65 4.0 



Note — 1978-79 data based on an N of 22 and 1979-80 on 20 

Furthermore, respondents could cite in response to the first open-ended item 
benefits of evaluation. These were, in order of frequency: 



EKLC 
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At the same time we conducted the 1979-80 Evalxiation of Evaluation Services 
we were developing our goals for evaluation. Since our perspective was 
one of using evaluation dita, our goals were oriented toward the ultimate 
goal of action on evaluation findings. We came to hypothesise , in our 
setting anyway y three key variables related to action: Quality of our 
working relationships , the selection of key findings , and joint planning 
of action In response to findings by evaluation and project staff. Unique 
to our setting because of the nature of our finding were two other goals: 
Spending an appropriate amount of time on each evaluation and getting our 
reports out on time. 

The following page contains our goal statements and a schematic showing 
their Interrelationship. This page Is followed by a report — goal by 
goal — of each goals* operatlonalization into an objective and the 
results of our 1979-80 survey. Based on this work we are planning a new 
survey y for 1980-81. Appendix C contains a draft of this instrument. 

Three caveats are necessary: 



1. 1979-80 data are based on eleven programs the majority of which 
were in one program administrative area, and all under one 
evaluation manager. We do not expect our data to be as flatter- 
ing this year as we will be assessing as many as forty programs 

in several program administrative areas and under several evaluation 
managers . 

2. Because we have no resources to conduct this study, i.e., it is 
an add on task, we had to use a survey form. Far more desirable 
would have been independent interviews supplemented by content 
analysis of our work. 

3. Because of increased institutional demands on our time (5%) 
and the inflation in our wages and supplies (15%), we will have 
at best 80% of our 1979-80 resources available to do evaluation 
work in 1980-81. 
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GOALS FOR EVALUATION 



(1) Evaluation Services staff account for each program* s share of evalua- 
tion time by working the appropriate nusaber of hours on that project 
by the project end date. 

(2) Evaluation Services staff complete the final evalxiation report (includ- 
ing review and release) by the project end date. 

(3) Evaluation Services staff develop positive and productive working 
relationships with program staff. 

(4) Eivaluation Services reports findings highlighting those which could 
be the basis for action by prograiu staff to Increase the quantity 
and quality of service they provide. 

(5) Evaluation Services assists program staff in planning action based 
on evaluation findings, action which. If implemented, would increase 
the quality and quantity of service per unit of program cost. 

(6) Program staff act on relevant evaluation findings to increase the 
quantity and quality of services per unit of program cost. 



EOtf THE GOALS FOR EVALUATION RELATE TO USING EVALUATION 
DATA IN THE PROCESS OF IMPR0T2IG PROGRAMS 



(1) Time on Task 



(2) Timely Reporting 



(3) Working Relationships 

(4) Selection of Key Findings-^ (6) ACTION 

(5) Joint Planning of Action 



• O • 



DfllEDIATE GOALS 



Evaluation Servioee eixtff account fov each program's share of evaluation 
titm by working the appropriate maiber of hours on that project by the pro- 
gect end date. 

Objective 1 ; "Evaluation Services . . . (same as goal) ... as documented 
on the Weekly Time Accounting System". 

1979-80 Data: We met this goal for each one of our eleven evaluations. 

Evaluation Services staff compUte the final evaluation report (including 
review and release) by the progeat end date. 

Objective 2 ; "Evaluation Services. .. (same as goal)... as documented 

by the release date on the cover memo to final reports" - 

1979-80 Data: Eight of eleven reports were released on or before 

their due date; the other three reports were released 
1.5, 2.0, and 3.5 weeks late. A twelfth report, a 
second Adult Education report (program statistics) 
could not be completed until four months beyond its 
scheduled date because of missing documentation. 

Evaluation Services staff develop positive and productive working relation- 
ships with program staff. 

Objective 3.1: Evaluation Ter^^ces staff will have better working rela- 
tionships with ESC-20 project staff. Accomplishment 
of this objective will be Indexed by at least two- 
thirds of the judgments of project staff being "agree" 
or "strongly agree" (4 or 5 on a 5-point scale) on 
items of the Evaltiating Evaluation services survey 
addressing such topics as Evaluation Services' (a) 
lack of negativism (b) understanding of programs, 
staff and constraints, and (c) working with program 
staff to (1) find out what they want, (11) keep them 
Informed of evaluation's program, and (ill) interpret 
and use findings.. 



1979-80 Data : Based on the responses of 27 staff (17 special educa- 
tion consultants, 4 non-special education consultants, 
and 6 project managers /coordinators) across the five 
items in Part II addressing this objective 89% of the 
judgments made were "4" or "5" and the mean judgment 
was 4.4. 



Objective 3,2; Evaluation Services staff spend, on the average evalua- 
tion, 52 of their time in interaction with project 
staff as documented by code 07 on the Weekl/ Time 
Accounting Form. 

1979-80 Data ; Data collection did not start until 1980-81 



IHTERMEDIATE GOALS 



Evdltdatum Sexn>icea reports findings highMghUng those which could be the 
hmis for action by program staff to increase the quantity and quality of 
service they provide* 

Objective 4; "Valuation Services... (same as goal)... as evidenced 
by at least two- thirds of the judgments of project 
staff being "agree" or "strongly agree" to items on 
the Evaltxating Evaluation Services survey addressing 
our written and verbal highlighting of findings that 
could lead to Improvement in servicing. 

1979-80 Data: Based on the responses of 27 staff (17 special educa- 
tion consultants, 4 non-special education consultants, 
and 6 project managers/coordinators) 922 of the judg- 
nents made on t\e two items addressing highlighting 
findings were "agree" or "strongly agree" and the 
mean judgment was 4.3. 



EoaZuaticn Services assists program staff in planning action based on EvaU 
uaticn findings^ action which^ would increase the quality and quantity of 
service per latit of pr o yrof n cost. 



"Evaluation Services assists program staff in planning 
action based on evaluation findings; action which, if 
l»»^leatent€d would increase the quality and quantity of 
service per unit of program cost as evidenced by at 
least two-thirds of the judgments of project staff 
being "agree" or "strongly agree" to items on the 
Evaluating Evaluation Services survey addressing (a) 
positive Impact of evaluation, (b) necessity of eval- 
uation, ajjjj (c) improvements tc their programs. 

i9Z?=§5-5^: Based the responses of 27 staff (17 special educa- 
tion consultants, 4 non-special education consultants, 
and 6 project managers/coordinators) 71% of the judg- 
flieflts made on the two items addressing evaluation's 
positive impact and necessity were "agree" or "strongly 
agree ^j^^ mean judgment was 4.0. 



ULTIMATE GOAL 



Progx>am staff Oot ^ ^Uvant evaluation findings to increase the quantity 
and quality of ^Bx^ceS P^^ unit of program cost. 



not addressed in 1979-80. 
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Appendix A: 1978-79 Survey Instrument 



The aiost important thing we ought to change about the way programs are eval- 
uated Is: 



2. 




The major benefits I have observed from the evaluation of special education 
programs are: 
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4. 



"The presence of Evaluation Service has had a positive impact in improving 
ESC-20 Special Education programs with which I am familiar." (Circle One) 



Strongly 
Di sagree 



Disagree 



Neutral 



Agree 



Stongly 
Agree 



5. 



Rank order the six services provided by Evaluation Services listed below 
usTng the lines under 5a. Rank the most important service-1. the se- 
cond inost-2v and so forth. Then rate each service by circling the num- 
ber under 5b. 



■1 
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5a 



Not 
Useful 



A. As a sounding 

board for ideas 
V aad/or a source 
v. of i4eas 

Assistance 
with - 
develt^ing 
Obiectives 

Needs Assess- 

0. Assessing the 
accaaH>li shment 
of project 
objectives 



1 
1 



Assistance in 
formulating recom- 
aendadons to Improve 
programs 



Workshop 
evaluations 



5b 



Of 
Minimal 
Use 
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Of Some 
Use 



3 
3 



20 



Definitely 
Useful 



4 
4 



Very 
Useful 



5 
5 



^pendix B.: 1979-80 Survey Instrument 



PA2T 1: Please con^lete the following sentences. 



The most important benefits or services I receive froa Evaluation 
Services, are • • • 



The greatest problems I experience in working with Evaluation Services 
are • • • 



Evaluation Services really ought to change the way it • . . 




^AitT TT* The following Items assess Evaluation Sexvices' accoiaplishaeat of 
thit, f oUowtog goal: Evaltdotian Sertrlces ' staff develop positive 
and productive workinff relaticnships with pTogvam staff. For 
eacli of tlie following statements, circle the ri^^onse that indi- 
cates how evaluation staff have worlced with. you. 



^Vmluat^l*'^ Services staff 
^espons^l^ for evaluation 

^ vfOt^ with me to find 
out what kind of in- 
formation I want. 

^% loe^ M informed of 
tine progress of the 
eV«^natlon. 

c« present findings in 
^ positive^ construe- 
ti-^ vay. 

vcrk with me 

CO Interpret and use 

evaltiatlon findings. 

ft, TB^pt e postlve effort 
^ understand wf pro— 
g^am^ my goals» and 
the constraints Z am 
facing. 



1 

Strongly 
Disagree 



Use this scale to race items a-e 



2 

Disagree 



3 

Neutral 



4 

Agree 



5 

Strongly- 
Agree 



EAST lH: These itoas assess Evaluation Services' accoxnpllshinent of the 
follovdbig goal: Evaluation Seivices reports findings • . . 
io increase quarttitt; and quaZitif of service . . . by amounta 
coHTTiensiirat^ with the cost of evaluation. 





1 

Strongly 
disagree 


2 

Disagree 


3 

Neutral 


4 

Agree 


c 
D 

Strongly 
Agree 


a. 


The presence of 
Evaluation Services 
has had a positive 
Impact in Improving 
ESC-20 programs with 
vhlch X am familiar* 


1 


2 

• 


3 


A" 


5 




Evaluation Services* 
written reports high- 
ll^t those findings 
that could lead to 
Improvement of our 
services. 


1 


2 


3 


4 


5 




Evaluation Services* ^ 
. staff highlight In dis- 
cussions those findings 
that could lead to Im^ 
provement of our services. 


2 


3 


4 


5 


d. 


Based on Tay escperlences 
vith Evalxaatlon Services » 
I helleve evaluation is 
a necessary and impor- 
tant service. 


1 


2 


3 


4 


5 
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Part I: 



Appendix C: Draft of 1980-81 Survey Instrument 



As a sounding board for 
Ideas or a source of Ideas 

Assistance with proposal 
objectives and evaluation sections 

Needs Assessments 



ANY OTHERS?: 



Assessing accomplishment 
of project objectives 

Program Improvement 
Program documentation 





We're hard to work with 


Our work Is of poor quality 


Me lack knowledge 


Our work Is too late 


Wte're not there when you need us 


We create too much paperwork 


Me do too little work 


We take too much time 


Our work is hard to understand 


Our work Is not relevant 


ANY OTHERS?: 





EVALUATION SERVICES REALLY OUGHT TO CHANGE THE WAY IT... 




AGREE 



STRONGLY 
AGREE 



Part II: This section contains In Italics the goals of Evaluation Services and beneath 
each goal ItenCs) assessing that goal. For each Item circle the response that 
Indicates hov strongly you agree or disagree with the Item. 

Ei}aZuacb£cn Services staff devetcfp positive and 
produotCve working relationships with proc ram 

^taff. * STRONGLY DISAGREE NEUTRAL 

DISAGREE 

Evaluation Services staff work with me to 

find out what kind of Information I want. 1 2 3 4 

Evaluation Services staff make a positive 
effort to understand my program, my goals , 

and the constraints I am facing. 1 2 3 4 

Evaluation Services staff keep me Informed 

of the progress of my evaluation. 1 2 3 4 

Evaluation Services staff present findings 

In a positive, constructive way. 1 2 3 4 

B(}aliiation Services reports findings high- 
lighting l^hose which could he the basis for 

Evaluation Services *s written reports hlgh- 
Ugiht those findings that could lead to 

liq>roveaient of our services* 1 2 3 4 

Evaluation Services highlight In discussions 
those findings that could lead to improvement 

of our services. 1 2 3 4 

EvaUurtion Seri/ices assists program staff in 
planning action based on ei^lnation findings* 

Evaluation Services works with me to inter- 
pret and use evaluation findings. 1 2 3 4 

The presence of Evaluation Services has had 
a positive ii^>act in improving ESC- 20 pro- 
grams with which I am familiar. 1 2 3 4 

Based on my experiences with Evaluation Ser- 
vices, I believe evaluation la a necessary and 
iBq>ortant service. 

Program staff act on relevant evaluation findings. . . 



Circle the item which best characterizes the usefulness of evaluation 
Information to you. 

!• I cannot act on the evaluation information I receive to improve 
Wf program. 

2. I cstn act on evaluation information to make Improvements in my 
services, but the;;e are definitely minor improvements. 

I can act on evaluation information to make improvements in my ser- 
vices, but these improvements, tihile Important, are not critical. 

I can act on evaluation information to make major improvements in 
Wf delivery of service. 



3. 



4. 
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